Features compared

Jürgen Schneider

25 September 2023

Why even bother?



  • specialized software organizes screening task
    idea: ready-to-use setup, tailored at screening for reviews



  • save time
    sort documents by relevance via machine learning

Software options

(Burgard & Bittermann, 2023)

Active learning



Example: SVM (support vector machines) used by Rayyan:

  1. manually reviewed references word combinations that differentiate between included and excluded
  2. predictive model computes inclusion probabilities to the remaining records


ASReview offers several algorithms to choose from



Output:
ASReview sorts the remaining records
Rayyan symbolizes inclusion probability as stars (and sorts)

How good does it work?



Stopping rules:

  • pragmatic: after …% of screened documents
  • heuristic: after … consecutive irrelevant documents
  • sampling: tool estimates to have found …% of relevant documents



Metrics:

  • Recall: Share of relevant papers found
    89 of 100 relevant papers were found: recall of .89

  • WSS: Work saved over sampling (for a certain recall)
    After screening 70% of documents, I found 100% of relevant records, therefore saved 30% of work

(Scherhag & Burgard, 2023)

How good does it work?

(Scherhag & Burgard, 2023)

How good does it work?

(Scherhag & Burgard, 2023)

How good does it work?



  • ASReview outperforms Rayyan

  • Stopping after …% most reliable

  • Combination of stopping rules
    stopping after min. 80% screened AND 50 consecutive irrelevant

Practical features

feature ASReview Rayyan
Free Yes (Yes) Free version available
Open Source Yes No
Platform User-unfriendly local installation Browser-based or mobile version
Data remains on local machine Yes No

Practical features

feature ASReview Rayyan
Detecting duplicates No Yes
Two or more coders possible No Yes
# of docs to be coded before sorting algorithm can run 2 (1 exclude, 1 include) 50
Blinding authors possible Yes No
Keywords for include/exclude to be highlighted in abstracts No Yes (each coder)
Specific feature to code exclusion reason No (needs to be typed in manually in the notes field for every doc) Yes (once typed in, can be coded with other docs)
Feature to code labels No Yes (once typed in, can be coded with other docs)
Export of coded results in an easy to use/read form Yes No (but I can share a script for that)

Hands-on



Script for Rayyan


You’ll only need to change things in the lines with comments mentioning “DO:”


library(tidyverse)
library(rio)

freq_screened <- rio::import(file=here("[path to]/articles.csv")) # DO: INSERT PATH

freq_screened <- freq_screened %>%
  dplyr::mutate(excl_reason = str_extract(.$notes, "RAYYAN-EXCLUSION-REASONS:.*?(\\||$)"), # extract exclusion reasons
                labels = str_extract(.$notes, "RAYYAN-LABELS:.*?(\\||$)"),       # extract labels coded
                decision_c1 = str_extract(.$notes, "Jürgen\"\"=>\"\".*?\"\""),    # extract decision from coder 1, DO: CHANGE NAME!
                decision_c2 = str_extract(.$notes, "Juergen\"\"=>\"\".*?\"\"")) %>% # extract decision from coder 2, DO: CHANGE NAME!
  # remove prefixes or recode
  dplyr::mutate(excl_reason = str_remove_all(excl_reason, "RAYYAN-EXCLUSION-REASONS:\\s"),
                labels = str_remove_all(labels, "RAYYAN-LABELS:\\s|\\s\\|"),
                decision_c1 = case_when(                                      
                  str_detect(decision_c1, "Excluded") ~ "Excluded",           
                  str_detect(decision_c1, "Included") ~ "Included",           
                  str_detect(decision_c1, "Maybe") ~ "Maybe"),                
                decision_c2 = case_when(                                      
                  str_detect(decision_c2, "Excluded") ~ "Excluded",           
                  str_detect(decision_c2, "Included") ~ "Included",           
                  str_detect(decision_c2, "Maybe") ~ "Maybe"),
                agreement = case_when(        # is there agreement on the decision?
                  decision_c1 == decision_c2 ~ 1,
                  TRUE ~ 0)
  )

References

Burgard, T., & Bittermann, A. (2023). Reducing Literature Screening Workload With Machine Learning: A Systematic Review of Tools and Their Performance. Zeitschrift Für Psychologie, 231(1), 3–15. https://doi.org/10.1027/2151-2604/a000509
Scherhag, J., & Burgard, T. (2023). Performance of Semi-Automated Screening Using Rayyan and ASReview: A Retrospective Analysis of Potential Work Reduction and Different Stopping Rules. https://doi.org/10.23668/PSYCHARCHIVES.12843